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November  19,  1992 

The  Honorable  John  Conyers 
Chairman,  Legislation  and 
National  Security  Subcommittee 
Committee  on  Government  Operations 
House  of  Representatives 

Dear  Mr.  Chairman: 

In  response  to  your  request,  we  assessed  the  results  of  the  AN/BSY-1 
operational  evaluation,  including  testing  limitations  and  their  impact  on  the 
evaluation.  We  also  attempted  to  determine  whether  the  operational 
evaluation  was  conducted  independent  of  the  AN/BSY-1  program  office  and 
its  contractors  and  reviewed  the  Director,  Operational  Test  and 
Evaluation’s  (dot&e)  oversight  of  the  test.  This  report  is  a  follow-on  to  our 
January  1992  report.' 
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The  Navy  is  equipping  improved  Los  Angeles  class  nuclear-powered  attack 
submarines  (SSN-6881)^  with  the  new  AN/BSY-1  combat  system.  The 
AN/BSY-1  program  evolved  in  1985  as  a  result  of  the  restructuring  of  the 
Submarine  Advanced  Combat  System  program.  AN/BSY-1  is  designed  to 
support  SSN-688Is  in  conducting  combat  operations  in  antisubmarine, 
antisurface,  strike,  mine,  and  other  warfare  missions.  Thus,  a  fully  capable 
combat  system,  successfully  developed  and  tested,  is  critical  to  SSN-688Is 
accomplishing  their  missions. 

The  Navy  contracted  with  the  International  Business  Machines  Corporation 
to  concurrently  develop  and  produce  23  AN/BSY-1  systems,  3  maintenance 
and  team  trainers,  and  a  software  maintenance  facility.  In  May  1987,  the 
first  AN/BSY-1  was  delivered  to  the  Navy  for  installation  in  the  SSN-751 , 
which  was  later  used  for  the  AN/BSY-1  operational  evaluation.  As  of 
September  1992,  the  International  Business  Machines  Corporation  had 
delivered  22  systems  to  the  Navy,  14  of  which  have  been  installed  in  newly 
constructed  SSN-688Is.  As  of  September  30,  1992,  total  development  and 
production  cost  of  the  AN/BSY-1  was  estimated  at  $3.8  billion. 


'Navy  Acquisition:  Development  of  the  AN/BSY-1  Combat  System  (GAO/NSIAD-92-50,  Jan.  31 ,  1992). 


^The  Navy  refers  to  the  U.S.S.  San  Juan  (SSN-75 1 )  and  subsequently  constructed  submarines  of  the 
class  as  the  Improved  SSN-688.  In  addition  to  AN/BSY- 1 ,  these  submarines  have  all  the  improvements 
added  to  previous  submarines  of  the  class,  the  ability  for  increased  under-ice  operations,  and  several 
new  quieting  modiflcations. 
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As  the  Navy’s  independent  agent  for  operationeil  test  and  evaluation,  the 
Commander,  Operational  Test  and  Evaluation  Force,  is  responsible  for 
planning,  conducting,  and  reporting  the  results  of  all  operational  tests  and 
evaluations,  without  the  participation  and  influence  of  the  developing 
agency  and  its  contractors,  and  providing  a  recommendation  regarding  the 
weapon  system’s  readiness  for  fleet  introduction  or  need  for  further 
testing.  Short  of  war,  operational  test  and  evaluation  is  the  primary  method 
used  to  predict  how  a  weapon  system  is  likely  to  perform  in  combat. 
Operational  evaluation,  the  final  phase  of  operational  test  and  evaluation,  is 
an  overall  assessment  to  determine  whether  the  system  can  perform  its 
mission  as  intended  (operational  effectiveness)  and  whether  it  will  be 
reliable,  maintainable,  and  operationally  available  when  deployed 
(operational  suitability). 


Results  in  Brief 


The  Operational  Test  and  Evaluation  Force  conducted  the  AN/BSY-1 
operational  evaluation  from  November  1 , 1990,  to  May  17, 1991 .  The 
Commander  concluded  that  the  AN/BSY-1  was  operationally  effective  and 
suitable  and  recommended  it  be  approved  for  introduction  into  the  fleet. 
We  do  not  disagree  with  the  Commander,  Operational  Test  and  Evaluation 
Force’s  conclusions  or  the  recommendation  that  AN/BSY-1  be  approved 
for  fleet  use,  considering  that  (1)  23  of  the  27  critical  operational  issues® 
examined  were  satisfactorily  demonstrated;  (2)  AN/BSY-1  is  a  mature 
program  with  no  significant  programmatic  decisions  remaining  (all 
systems,  trainers,  and  the  software  maintenance  facility  have  been 
approved  for  production  and  all  contracts  awarded);  (3)  96  percent  of  the 
systems  and  all  trainers  have  been  delivered  to  the  Navy;  and  (4)  initial 
operating  capability  has  been  achieved.  However,  we  note  the  following: 

Two  AN/BSY-1  critical  operational  issues  (reliability  and  maintainability) 
were  unsatisfactory.  System  failures  decreased  AN/BSY-1  reliability.  The 
failures  were  not  corrected  in  the  required  time.  Changes  are  being  made 
that  are  expected  to  correct  these  deficiencies. 

Limitations  resulted  in  incomplete  demonstration  of  two  critical 
operational  issues  (weapon  employment^  and  navigation)  and  uiurealistic 
operational  testing.  We  recognize  that  operational  testing  cannot  always  be 
done  in  an  environment  that  totally  represents  all  operational  conditions 


^Critical  operational  issues  are  key  efTectiveness  and  suitability  issues  that  must  be  demonstrated 
during  ot)erational  test  and  evaluation  to  determine  the  system's  capability  to  perform  its  mission. 

*This  issue  involved  the  submarine-launched  mobile  nune.  Navy  and  Department  of  Defense  ofTicials 
determined  that  the  mine  was  inappropriately  included  in  the  operational  evaluation  report.  As  a  result, 
it  is  not  discussed  in  our  report. 
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and  that  the  Navy  has  taken  steps  to  improve  the  realism  of  operational  test 
and  evaluation.  Nevertheless,  reducing  these  limitations  can  be  crucial  to 
identifying  and  correcting  problen\s  before  a  weapon  system  is  produced 
and  deployed. 

•  We  could  not  reach  a  conclusion  on  the  independence  of  the  operational 
evaluation.  SSN-751  records  show  AN/BSY-1  program  and  contractor 
personnel  were  on  board  the  submarine  during  dedicated  testing  days.  The 
Navy  could  furnish  no  documentation  that  these  personnel  left  the 
submarine  before  AN/BSY-1  operational  evaluation  began. 

•  Based  on  limited  observations,  DOT&E  determined  the  AN/BSY- 1 
operational  evaluation  was  adequate.  However,  because  dot&e  does  not 
establish  the  number  of  on-site  visits  required,  we  could  not  assess  how 
effectively  dot&e  carried  out  its  responsibilities. 


AN/BSY-1  ReUability 
and  Maintainability 
Were  Unsatisfactoiy 


Commander,  Operational  Test  and  Evaluation  Force  Instruction  3960.  IG 
requires  all  critical  operational  issues  be  satisfactorily  resolved  before 
recommending  a  system  for  introduction  into  the  fleet.  The  AN/BSY-1 
operational  evaluation  report  shows  23  of  the  27  critical  operational  issues 
examined  were  satisfactorily  demonstrated,  but  two  suitability  issues 
(reliability  and  maintainabihty)  were  unsatisfactory.  The  system  was 
deemed  unreliable  because  operators  had  difficulty  with  the  system’s 
acoustic  software.  Primarily,  AN/BSY-1  ’s  maintainability  was  degraded 
because  of  excessive  troubleshooting  time.  However,  the  operational 
evaluation  showed  AN/BSY-1  would  be  available  over  96  percent  of  the 
time.  Because  operational  availability  exceeded  operational  requirements, 
the  Commander  concluded  that  AN/BSY- I’s  operational  availability 
outweighed  the  reliability  and  maintainability  issues  and  that,  from  an 
operational  standpoint,  AN/BSY-1  was  suitable.  Consequently,  the 
Commander  recommended  AN/BSY-1  be  approved  for  fleet  use.  Significant 
improvements  are  being  installed  to  correct  the  reliability  and 
maintainability  issues. 


Systein  Failures  Decreased  Reliability  is  the  probability  that  AN/BSY-I  will  continuously  operate  during 
AN/BSY-1  Reliability  a  mission  without  critical  failures  or  errors.  The  measurement  is  expressed 

as  the  relation  of  total  operating  time  to  the  number  of  system  failures  or 
errors.  AN/BSY- 1  did  not  meet  5  of  1 1  hardware  and  software  reliability 
goals. 

AN/BSY-1  hardware  reliability,  expressed  as  mean  time  between  failure, 
was  assessed  for  two  configurations;  (1)  with  all  system  fimctions  fully 
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System  Failures  Were  Not 
Corrected  Within  Required 
Time 


operational  and  (2)  with  only  those  functions  required  to  preempt  enemy 
attacks  and  allow  the  submarine  to  operate  safely.  AN/BSY-1  ’s 
demonstrated  mean  time  between  failure  for  the  fully  operational 
configuration  exceeded  its  goal  by  76  percent,  but  the  limited,  self-defense 
goal  was  7  percent  less  than  the  Test  and  Evaluation  Master  Plzm  required. 

AN/BSY-1  software  longevity,  expressed  as  mean  time  between  error,  was 
computed  for  various  actions  required  to  resume  normal  computer 
operations.  Overall,  AN/BSY-1  software  longevity  was  determined  to  be 
unsatisfactory  because  criteria  in  four  of  nine  measured  areas  were  not 
met.  For  example,  AN/BSY-l’s  demonstrated  mean  time  between  error  for 
warm  and  hot  starts  for  acoustic  functions  were  28  percent  and  37  percent, 
respectively,  below  established  goals. 

System  failures  were  corrected  by  reloading  or  reassigning  its  software.  A 
warm  start  required  the  operator  to  download  and  then  reset  the  computer 
to  resume  system  operations.  All  data  was  saved  except  incoming  data  lost 
during  the  interval  between  error  and  completion  of  the  recovery  action.  A 
hot  start  allowed  the  operator  to  reassign  the  software  or  micro-code 
programs  without  degrading  acoustic  performance.  Reassigning  the 
software  was  the  preferred  recovery  action.  However,  due  to  its 
complexity,  sonar  operators  often  bypassed  this  procedure  to  save  time. 


Several  measurements  were  computed  to  assess  the  crew’s  ability  to  keep 
the  system’s  equipment  in  specified  operational  condition.  For  hardware, 
the  measurement  is  expressed  as  mean  time  to  repair,  or  the  average 
elapsed  time  between  problem  detection  and  restoration  of  satisfactory 
operations.  It  includes  the  average  time  to  isolate  the  defective  part,  obtain 
the  appropriate  shipboard  replacement  part,  disassemble  and  replace  the 
defective  part,  reassemble,  align,  and  check  out  the  repair. 

The  mean  tirtie  to  repair  for  1 2  part  failures  was  176  minutes,  or 
85  percent  longer  than  required  to  meet  the  operational  requirement  for 
hardware  maintainability.  Two  of  the  12  parts  were  not  repaired  because 
replacement  parts  were  not  carried  on  board.  For  these  parts,  the  test 
measured  only  the  time  required  to  detect  and  isolate  the  defects. 

About  78  percent  of  the  total  repair  time  was  used  to  identify  the  cause  of 
system  failures.  Although  AN/BSY-1  has  a  built-in  error-detection  system  to 
identify  problem  areas  and  provide  instructions  to  correct  them,  the 
built-in  test  did  not  effectively  detect  and  isolate  the  problems.  According 
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to  the  report,  troubleshooting  instructions  were  voluminous,  did  not  allow 
for  finding  the  correct  work-around  procedure,  and  did  not  help  the 
operator  to  understand  the  system  response  in  the  short  amount  of 
decision  time  allowed  during  a  system  casualty. 


Planned  Changes  Expected  The  Navy  expects  two  changes  to  correct  system  deficiencies  detected 
to  Correct  Deficiencies  during  AN/BSY-1  technical  and  operational  evaluations.  The  first  change 

provides  software  and  hardware  modifications  to  correct  281  technical 
evaluation  deficiencies.  For  example,  the  change  will  improve  the  speed 
and  performance  of  the  AN/UYK-43  computer  and  enhance  processing.  As 
of  September  30,  1992,  the  change  had  been  installed  on  1 1  systems.  Five 
additional  systems  are  expected  to  be  upgraded  by  April  1993.  The 
remaining  nine  systems  will  be  delivered  to  the  ship  with  the  change 
already  installed.  Because  identical  problems  were  noted  during  the 
AN/BSY-1  operational  evaluation,  some  technical  evaluation  improvements 
will  eliminate  the  operational  evaluation  deficiencies. 

The  second  change  is  an  acoustic  software  modification  designed  to 
address  60  system  deficiencies  observed  during  operational  evaluation. 
This  change  will  also  correct  the  two  unsatisfactory  critical  operational 
issues.  Improvements,  for  example,  will  be  made  to  correct  software 
error-detection  and  longevity  problems  and  to  reduce  the  number  of  false 
alarms  and  reconfiguration  management  actions.  This  change,  estimated  to 
cost  about  $2.1  million,  is  being  installed  on  all  systems.  As  of 
September  30, 1992,  this  change  had  been  installed  in  10  systems; 

6  additional  systems  are  expected  to  be  upgraded  by  May  1993.  The  nine 
remaining  systems  will  be  delivered  to  the  ship  with  the  change  already 
installed.  Verification  that  the  deficiencies  have  been  corrected  will  be 
made  during  follow-on  test  and  evaluation. 


resting  Conditions  and 
Inadequate  Equipment 
Effected  Operational 
Evaluation 


Department  of  Defense  test  and  evaluation  policy  requires  that,  to  the 
extent  practicable,  (1)  testing  be  done  in  geographical  settings  and  under 
weather  conditions  that  are  representative  of  the  operating  environment 
and  (2)  equipment  used  be  representative  of  the  threat.  Since  the  1970s, 
we  have  issued  numerous  repoits  on  the  Department  of  Defense’s  need  to 
improve  the  quantity  and  quality  of  operational  test  and  evaluation.  In 
December  1986,  for  example,  we  reported®  that  the  usefulness  of 
operational  test  and  evaluation  in  estimating  a  weapon  system’s 


^Weapon  Performance:  Operational  Test  and  Evaluation  Can  Contribute  More  to  Decisionmakinf! 
(GAO/NSIAD.87-57,  Dec.  23.  1986). 


Pages 


6AO/NS1AD-93-81  Navy  Acquisition 


B-220298 


Limitations  Result  in 
Incomplete  and  Unrealistic 
Operational  Testing 


performance  has  been  limited  because  of  long-standing  problems  such  as 
test  sites  not  being  representative  of  the  operating  environment  or  test 
resources  not  always  being  available  or  adequate. 

Our  assessment  of  the  AN/BSY-1  operational  evaluation  shows  that  these 
problems  still  exist.  During  the  operational  evaluation,  the  test 
environment  precluded  the  Operational  Test  and  Evaluation  Force  from 
fully  assessing  all  navigation  issues.  In  other  instances,  testing  limitations 
affected  the  operational  evaluation.  None  of  these  limitations,  however, 
prevented  the  Commander,  Operational  Test  and  Evaluation  Force,  from 
formulating  conclusions  regarding  AN/BSY-l’s  operational  effectiveness 
and  suitability. 


One  critical  operational  issue— advanced  under-ice  navigation— was  not 
satisfactorily  demonstrated  due  to  testing  limitations.  In  addition,  not  all 
test  equipment  used  for  the  AN/BSY-1  operational  evaluation  were 
adequate.  Noise  simulators  that  emitted  threat  submarine  sounds  into  the 
water  were  unrealistic.  Further,  the  weapon  range  was  too  small  to  allow 
realistic  testing  of  AN/BSY-l’s  ability  to  perform,  particularly  during 
multiple-ship  engagements.  In  addition,  the  poor  reliability  of  torpedoes 
hampered  evaluation  of  AN/BSY-1  operational  effectiveness  in  torpedo 
engagements.  According  to  DOT&E,  programs  are  available  to  correct  these 
deficiencies,  but  funding  is  inadequate. 

The  capability  to  navigate  under  ice  using  the  forward  look  sonar  was  not 
completely  tested.  Although  AN/BSY-1  demonstrated  the  capability  to 
safely  operate  while  piloting  and  to  detect  and  perform  other  maneuvers  in 
an  area  where  the  ice  cover  varied  in  thickness  emd  size,  a  pack  ice 
environment^  was  not  available  due  to  the  scheduling  of  the  operational 
evaluation.  Advanced  under-ice  navigation  will  be  tested  during  the  Navy’s 
1993  Ice  Exercises. 

During  testing  on  the  range  and  the  ocean,  a  noise  augmentation  unit, 
which  is  designed  to  simulate  the  radiated  noises  and  tones  of  threat 
submarines,  was  attached  to  submarines  supporting  the  test.  However, 
according  to  a  DOT&e  official,  the  noise  imit  ^d  not  always  emit  submarine 
noise  levels  that  were  representative  of  threat  submarines.  Because  the  imit 
does  not  accurately  replicate  threat  submarines,  we  believe  some 
operational  problems  may  not  emerge  until  AN/BSY-1  actually  performs  in 


pack  tee  envtroiunent  is  any  area  of  sea  ice  Utat  forms,  but  does  not  remain  attached  to  the  coast. 
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its  intended  operational  environment.  However,  according  to  an  AN/BSY-1 
program  official,  no  detrimental  effect  or  risks  exist  because  of  the 
difference  in  emitted  submarine  noise  patterns. 

The  Navy  uses  the  Atlantic  Undersea  Test  and  Evaluation  Center  weapons 
range,  a  deep-water  facility  with  fixed  hydrophones,  to  make  underwater 
noise  measurements,  test  and  calibrate  sonars,  and  provide  accurate 
tracking  data.  According  to  DOT&E,  the  tracking  range  is  too  small  to 
provide  the  diversity  required  for  operational  realism  because  sensor 
systems  have  become  more  advanced  and  threat  submarines  much  quieter. 
For  example,  the  DOT&E  staff  assistant  told  us  that,  with  the  AN/BSY-1 
submarine  and  the  threat  submarine  at  opposing  ends  of  the  range, 
AN/BSY-1  was  able  to  pick  up  the  target  submarine’s  acoustic  signature 
without  ever  moving.  As  a  result,  full  capability  of  AN/BSY-1  could  not  be 
tested  at  the  range.  Full  capability  of  the  system  was  tested  in  the  open 
ocean  and  successfully  demonstrated. 

AN/BSY-1  demonstrated  the  ability  to  effectively  launch  and  control 
torpedoes.  However,  56  percent  of  the  torpedoes  that  missed  the  target  did 
so  due  to  the  torpedoes’  failures.  Operational  Test  and  Evaluation  officials 
told  us  that  since  the  test  was  conducted  on  an  instrumented  range,  it  could 
determine  if  the  torpedoes  would  have  hit  their  target  had  they  continued 
on  course. 


Independent  Testing 
Was  Not  Fully 
Documented 


We  were  unable  to  determine  whether  AN/BSY-1  program  and  contractor 
personnel  were  improperly  involved  in  the  actual  tests,  data  analysis,  or 
other  operational  evaluation  activities  for  the  AN/BSY-1  combat  system. 
Personnel  not  needed  to  support  the  testing  agency’s  mission  are  not 
supposed  to  be  on  board  testing  units  during  testing.  Authorization  can  be 
granted  if  such  visits  are  necessary  to  support  the  test  submarine  or  other 
unit  requirements  or  provide  technical  assistance  if  the  operational  test 
director  requests  it.  However,  these  personnel  are  to  leave  the  testing  units 
before  testing  resumes.  While  testing  agency  and  program  officials  stated 
that  all  interaction  between  them  ceased  when  AN/BSY-1  was  certified  for 
operational  evaluation  in  October  1990,  SSN-751  records  show  AN/BSY-1 
program  and  contractor  personnel  were  on  board  the  submarine  during 
dedicated  testing  periods.  Sufficient  data  did  not  exist  or  was  not  available 
to  verify  if  the  operational  evaluation  was  ongoing  while  the  personnel 
were  on  the  submarine.  Nevertheless,  their  presence  creates  the 
appearance  of  a  conflict  of  interest.  For  example: 
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•  The  TOMAHAWK  missile  is  one  of  four  weapons  the  AN/BSY-1  combat 
system  launches.  In  November  1990,  the  Chief  of  Naval  Operations  granted 
the  Program  Executive  Officer  for  Submarine  Combat  and  Weapons 
Systems  a  waiver  for  the  horizontal  and  vertical  TOMAHAWK  missile  test 
launch  during  the  AN/BSY-1  operational  evaluation.  Because  testing  was 
deferred  after  the  AN/BSY-1  had  been  certified  as  ready  for  operationail 
evaluation,  the  AN/BSY-1  program  office  was  required  to  recertify  the 
TOMAHAWK  missile  for  the  operational  evaluation.  On  February  1,  1991,  a 
dedicated  testing  day,  AN/BSY-1  program.  International  Business 
Machines,  and  Naval  Undersea  Warfare  Center  personnel  were  on  board 
the  SSN-751  to  provide  an  operational  test  launch  of  the  missile  to  recertify 
it  for  the  AN/BSY-1  operational  evaluation.  Later  that  day,  the  AN/BSY-1 
operational  evaluation  resumed.  The  Navy  could  not  provide 
documentation  that  program  and  contractor  personnel  had  left  the 
submarine  before  testing  resumed. 

•  On  May  1 1 , 1 99 1 ,  another  dedicated  testing  day,  AN/BSY-1  program. 
International  Business  Machines,  and  Naval  Undersea  Warfare  Center 
personnel  were  on  board  the  SSN-751  to  replace  the  TB-23  towed  array 
and  repair  the  UYK-43  acoustic  computer-components  of  the  combat 
system.  The  equipment  had  experienced  problems  earlier  during  the 
operational  evaluation  that  the  ship’s  cr^w  were  unable  to  repair.  Because 
the  operational  test  director  prepared  formal  reports  reflecting  these 
problems,  the  AN/BSY-1  program  and  contractor  personnel  were  allowed 
to  begin  solving  the  problem.  Testing  resumed  the  same  day,  but  the  Navy 
could  not  provide  documentation  that  the  program  and  contractor 
personnel  had  left  the  submarine  before  testing  resumed. 

In  discussions  with  Operational  Test  and  Evaluation  Force  officials,  they 
told  us  that  a  personnel  transfer  was  conducted  between  the  demonstration 
test  and  the  operational  test  of  the  TOMAHAWK  missile.  They  further 
stated  that  only  authorized  operational  evaluation  participants  were  on 
board  the  submarine  during  testing,  otherwise  agency  reports  would  have 
shown  exceptions. 


Adequate  Documentation  for 
Some  Operational  Test  and 
Evaluation  Force  Activities 
Did  Not  Exist 


Operational  Test  and  Evaluation  Force  guidance  provides  that  documents 
comprising  the  project  case  file  be  forwarded  to  the  appropriate  record 
center  and  never  destroyed.  However,  the  Operational  Test  and  Evaluation 
Force  did  not  maintain  a  record  of  all  its  activities.  The  lack  of 
documentation  made  it  difficult  to  verify  actions  taken  during  the 
operational  evaluation,  especially  when  personnel  rotations  and 
retirements  are  involved.  For  example,  Operational  Test  and  Evaluation 
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DOT&E  Observation  of 
AN/BSY-1  Operational 
Evaluation  Was 
Linuted 


Force  guidance  states  that  the  operational  test  director  should  maintain  an 
on-scene,  running  account  of  the  operational  evaluation,  including 
discussions  at  meetings,  meeting  attendants,  unusual  events,  etc.  The 
purpose  of  the  journal  includes  (1)  serving  as  a  historical  record  for  future 
reference,  (2)  serving  as  a  source  for  answers  to  new  questions  about  the 
test,  and  (3)  substantiating  data  if  events  are  later  questioned.  Although  a 
journal  consisting  of  the  operational  test  director’s  and  other  participants’ 
hand-recorded  observations  was  available,  it  did  not  provide  a  complete 
chronological  record  of  the  project  or  its  participants.  One  reason  such 
documentation  is  important  is  because  the  operational  test  director,  who 
was  responsible  for  the  AN/BSY-1  operational  evaluation,  left  the  Navy  and 
subsequent  directors  did  not  have  firsthand  knowledge  of  the  operational 
evaluation. 


Concerned  that  operational  test  and  evaluation  was  not  receiving  the 
needed  emphasis  and  independent  oversight,  the  Congress  established  the 
Office  of  the  DOT&E  within  the  Office  of  the  Secretary  of  Defense.  The 
Director’s  responsibilities  include,  among  other  things,  (1)  prescribing 
policies  and  procedures  for  conducting  operational  test  and  evaluation, 

(2)  issuing  guidance  to  and  consulting  with  the  military  departments  on 
major  defense  acquisitions,  (3)  monitoring  and  reviewing  operational  test 
and  evaluation  in  the  Department  of  Defense,  (4)  analyzing  the  results  of 
operational  test  and  evaluation,  and  (5)  reporting  to  the  Secretary  of 
Defense  and  the  Senate  and  House  Committees  on  Armed  Services  and  on 
Appropriations  on  whether  the  test  and  evaluation  performed  was 
adequate. 

In  a  March  1987  report,’  we  identified  several  deficiencies  affecting 
DOT&E ’s  effectiveness  in  carrying  out  its  oversight  responsibilities. 
Specifically: 

DOT&E  appeared  to  be  making  only  limited  numbers  of  actual  on-site 
observations  of  operational  tests. 

DOT&E ’s  analysis  of  operational  testing  was  primarily  based  on  military 
service  test  reports  with  little  assessment  of  actual  test  results. 

DOT&E  had  not  provided  policy  and  procedural  guidance  or  maintained 
reliable  records  on  some  of  its  principal  activities. 


’Testing  Oversight'.  OperatioruU  Test  sund  Evaluation  Oversight  Improving  But  More  is  Needed 
{GAO/NSIAD-87-108BR,  Mar.  18,  1987). 
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In  June  1989,  during  testimony  on  the  adequacy  of  Department  of  Defense 
operational  test  and  evaluation,  we  noted  that  DOT&E  had  not  taken 
corrective  action  to  resolve  these  problems.  For  example,  DOT&E  had  not 
established  policy  and  procedures  or  maintained  rehable  records  on  test 
observations  and  had  not  developed  a  formal  system  to  document  how 
DOT&E  staff  spend  their  time. 

Our  assessment  of  DOT&E’s  oversight  activities  for  the  AN/BSY-1 
operational  evaluation  shows  improvement,  but  reaffirms  some  of  these 
long-standing  problems.  While  dot&e  had  reviewed  and  approved  the 
AN/BSY-1  Test  and  Evaluation  Master  Plan  and  the  Operational  Test  Plan, 
it  had  made  only  a  few  on-site  observations  of  the  AN/BSY-1  operational 
evaluation.  In  addition,  DOT&E  has  not  provided  policy  or  procedural 
guidance  addressing  the  number  of  on-site  visits  needed  to  properly 
monitor  operational  tests  and  evaluations.  Rather,  the  number  of  visits  was 
left  to  the  discretion  of  the  staff  assistants  and  their  supervisors. 

During  the  6-month  operational  evaluation,  over  250  tests  were  conducted 
to  resolve  the  27  critical  operational  issues.  However,  the  DOT&E  staff 
assistant  and  its  contractor,  the  Institute  for  Defense  Analyses,®  made  only 
four  visits  to  monitor  AN/BSY-1  testing.  During  two  visits,  torpedo 
operations  and  mine  detection  and  avoidance  vulnerability  testing  were 
conducted.  Records  indicate  that  no  testing  was  ongoing  during  the  other 
visits. 

In  deciding  which  tests  to  observe,  the  dot&e  staff  assistant  said 
professional  judgment  and  prior  development  history  are  considered.®  He 
also  said  that  visits  are  timed  to  coincide  with  the  most  critical  phases  of 
the  tests  so  dot&e  can  obtain  the  maximum  information. 

In  add^-essing  the  adequacy  of  a  test,  the  dot&e  staff  assistant  said  the  staff 
conducts  reviews  to  ensure  that  there  is  agreement  between  the 
requirements  documents,  the  Operational  Test  Plan,  Test  and  Evaluation 
Master  Plan,  and  raw  test  data.  DOT&E’s  fiscal  year  1991  annual  report  to 
the  congressional  committees  indicates  that  its  activity  for  1991  involved 
oversight  of  208  programs,  including  the  AN/BSY-1 .  DOT&E  concluded  that 


*The  Institute  for  Defense  Analyses  primarily  provides  studies,  analyses,  computer  software,  models, 
and  other  technical  or  analytical  support  for  policy  and  program  planning  and  management. 

®The  current  staff  assistant  is  not  the  individual  who  made  the  on-site  visits  to  observe  the  AN/BSY-1 
operational  evaluation;  the  former  staff  assistant  is  no  longer  with  the  Nav^',  having  retired  in  July 
1991. 
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the  total  scope  of  the  AN/BSY-1  operational  evaluation  was  adequate  and 
that  the  AN/BSY-1  was  operationally  effective  and  suitable. 

Recommendations 

We  reconunend  that  the  Secretary  of  Defense  direct  the  Director, 
Operational  Test  and  Evaluation,  to  formally  establish,  before  the  start  of  a 
system’s  operational  evaluation,  (1)  the  number  of  on-site  observations 
required  to  properly  monitor  operational  tests  and  evaluations  and 
(2)  which  tests  should  be  observed. 

As  agreed  with  your  office,  unless  you  publicly  announce  its  contents 
earlier,  we  plan  no  further  distribution  of  this  report  until  30  days  from  its 
issue  date.  At  that  time,  we  will  send  copies  to  the  appropriate 
congressional  committees,  the  Director  of  the  Office  of  Management  and 
Budget,  and  the  Secretaries  of  Defense  and  the  Navy.  We  wiU  also  make 
copies  available  to  others  upon  request. 

Please  contact  me  at  (202)  275*6504  if  you  or  your  staff  have  any 
questions  concerning  this  report.  Other  major  contributors  to  this  report 
are  listed  in  appendix  II. 

Sincerely  yours. 


Richard  Davis 
Director,  Navy  Issues 
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Scope  and  Methodology 


To  determine  the  status  and  results  of  the  AN/BSY-1  operational  test  and 
evaluation  and  limitations  to  the  test,  we  analyzed  the  operational 
evaluation  report  and  verified  its  accuracy  by  randomly  selecting  and 
reviewing  the  raw  data  (charts,  plots,  logs,  questionnaires,  and  technical 
and  tactical  documents)  collected  during  the  test.  We  also  reviewed  the 
Test  and  Evaluation  Master  Plan,  the  Operational  Test  Plan,  and 
administrative  reports.  We  compared  these  documents  with  the  operational 
evaluation  report  to  identify  inconsistencies. 

To  determine  w'hether  the  operational  evaluation  was  conducted 
independently  of  the  AN/BSY-1  program  office  and  developing  contractors, 
we  analyzed  documents  relating  to  the  control  of  test  data  and  visitors  on 
board  the  ship.  We  also  reviewed  the  Operational  Test  Director  Journal. 

In  assessing  dot&e’s  oversight  of  the  operational  evaluation,  we  reviewed 
applicable  laws  and  regulations  to  ascertain  specific  responsibilities. 
Further,  we  examined  the  DOT&E  Fiscal  Year  1991  Annual  Report  to 
Congress,  which  includes  an  assessment  on  the  adequacy  of  the  AN/BSY-1 
operational  evaluation. 

We  discussed  each  issue  with  officials  in  Washington,  D.C.,  at  the  Office  of 
the  Under  Secretary  of  Defense  for  Acquisition,  the  Office  of  the  Under 
Secretary  of  Defense  for  Operational  Test  and  Evaluation,  Program 
Executive  Office  for  Submarine  Combat  and  Weapons  Systems,  and  the 
Office  of  the  Director  of  Naval  Test  and  Evaluation  and  Technology 
Requirements.  We  also  discussed  these  issues  with  officials  at  the 
Operational  Test  and  Evaluation  Force,  Norfolk,  Virginia;  Naval  Undersea 
Warfare  Center,  Detachment  Hawaii,  Lualualei,  Hawaii;  and  Submarine 
Development  Squadron  Twelve,  Groton,  Connecticut. 

We  conducted  our  work  from  January  through  October  1992  in 
accordance  with  generally  accepted  government  auditing  standards.  At 
your  request,  we  did  not  obtain  official  agency  comments  on  the  report; 
however,  we  discussed  the  information  with  Department  of  Defense  and 
Navy  officials  and  incorporated  their  comments  as  appropriate.  Generally, 
they  concurred  with  our  findings  and  conclusions. 
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Major  Contributors  to  This  Report 


National  Security  and 
International  Affairs 
Division, 

Washington,  D.C. 


Norman  Rabkin.  Associate  Director 
FYederick  A  Bigden,  Assistant  Director 
FYances  W.  Scott,  Evaliiator-in-Charge 
Laura  B.  Jackson,  Senior  Evaluator 


(39445S) 
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